Visual representations of photo albums

ABSTRACT

Systems and methods for selecting representative photos for a photo album are described. An example computing device may include a display, a logic device, and a storage device storing instructions executable by the logic device to, for each known individual recognized in one or more photos of the album, determine an affinity score for that known individual indicating a connection of that known individual to a) a user of the computing device and b) a context corresponding to the album. The instructions may be further executable to select one or more known individuals based on the affinity score to be included in a subset of the known individuals recognized in one or more photos of the album, and select a representative photo of each of the subset of the known individuals to be included in one or more of a highlight photoset and a cover photo for the album.

BACKGROUND

Users may capture a plurality of images over time. Such images may beorganized into albums corresponding to a date/period of time, an event,or another grouping parameter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example user interface for a collection of photos andphoto albums.

FIG. 2 shows an example user interface for a photo album.

FIG. 3 is a flow chart of an example method for selecting representativephotos for a photo album.

FIG. 4 is a flow chart of an example method for collecting informationand determining connections between an individual and a) a user and b)an album context.

FIG. 5 shows example block diagrams of devices that may be queried forinformation to determine user relationships and album contexts.

FIG. 6 is a block diagram of an example computing system.

DETAILED DESCRIPTION

Photo albums may be generated manually (e.g., with a user selectingphotos for inclusion in the albums) or automatically (e.g., according todates and/or content of the photos). In either case, the photos of thealbum may be tied to a particular event or other context (e.g., date,subject matter, etc.). In order to navigate a listing of photo albums,each album may be represented by a cover photo, and a user may select acover photo or other selectable feature of the album in order to view athe album and/or a summary of the album (e.g., a subset of photos in thealbum or a truncated version of the album).

In some album generation systems, the cover photo and photos in thesummary may be manually selected by the user. In other systems (or as adefault action if the user does not provide a selection), the coverphoto and photos in the summary may be automatically selected based onsimple criteria (e.g., a first photo, earliest dated photo, etc.).However, manually selecting photos takes time and effort, particularlywhen working with multiple and/or large albums. In some cases, thislevel of effort is prohibitive, and the user never ends up selecting ameaningful cover/summary photo. In cases where selections are made, thephotos that are selected may be sub-optimal, due to the time/effortbarrier or for other reasons. Important photos may be left out.Selections may be repetitive, include too many photos, and/or includeless insignificant photos.

Whether through manual selection or other methods (e.g., automaticselection), sub-optimal selection of cover/summary photos diminishes thevalue of the album to the user. For example, the user may be lessemotionally engaged by the album. The cover/summary photos may not catchthe user's attention while the user is browsing albums. When presentedwith a sub-optimal cover/summary photo, the user may not even rememberthe event associated with the album.

The disclosure provides systems and methods for generating albumrepresentations including a cover photo and/or summary The cover photoand photos included in the summary may be selected based on anintelligent rule-based mechanism. For example, the value of the coverphoto and/or summary may be most closely tied to the individualscaptured in the photos. Accordingly, the disclosed rule-based mechanismmay include recognizing faces in the photos of an album as correspondingto known individuals, and determining 1) a connection between the userand the known individuals and/or 2) a connection between the knownindividuals and an event or other context of the album. Weights may beapplied to the connections in order to determine known individuals thatare most relevant to the album. As used herein, known individuals mayrefer to people corresponding to recognized faces in the photos.Individuals may also refer to animals (e.g., pets) having faces that arerecognized in the photos. Some of the processes described herein mayalso be applied to objects recognized in the photos (e.g., faces ofdolls, landmarks, and/or other objects of significance to the user),where the objects may be treated similarly to the recognized faces inthe photos.

Photos including the known individuals may be selected for the coverphoto and/or summary of the album. In some examples, the selected photosmay be altered (e.g., cropped) in order to focus on the recognized facesof the individuals when presenting the cover photo and/or summary to theuser. The rule-based mechanism and associated photo album representationare described in more detail below.

FIG. 1 shows an example user interface 100 for a collection of photos asdisplayed on a display device 102 (e.g., a display integrated withand/or coupled to a computing device). As shown, the user interface 100includes indicators and/or representations of a plurality of albums 104and a plurality of photos 106 associated with a user. For example, theillustrated representations of albums 104 may include selectable coverphotos for the albums (e.g., each selectable to view the photos of theassociated album and/or an album summary), and the representations ofthe photos 106 may include selectable thumbnails for the photos (e.g.,each selectable to view a larger/original sized version of theassociated photo). Any suitable representation or indicator for albumsand photos may be viewable in user interface 100.

In the illustrated example, the photos 106 are organized by date andrepresent photos captured by the user and/or uploaded by the user to anapplication associated with the user interface 100. In other examples,photos 106 may be organized in a different manner (e.g., by subject,file name, size, location, user-selected order, and/or other parameter).Some or all of the photos 106 may be included in one or more of thealbums 104, and additional photos or albums may be viewed by scrollingthe user interface side-to-side, up-and-down, or otherwise adjusting theview of the user interface (e.g., via user input such as swiping a touchscreen of the display device 102, scrolling a wheel of a mouse connectedto the display device, performing a gesture, and/or another inputmechanism).

Each album 104 may be represented by a cover photo for that album. Forexample, the illustrated album titles may be overlaid over (orpositioned to one side of) the cover photo for that album. The coverphoto may include a photo from that album. The cover photo may be anunaltered photo from the album or a cropped, resized, or otherwisealtered photo from the album representing the contents of the album. Thecover photo may also serve as a selectable user interface element ofuser interface 100. For example, selecting the cover photo of a givenalbum (e.g., by tapping, clicking, performing a selection gesture,and/or otherwise directing user input toward the cover photo) may causethe given album to be selected for viewing. An example view of aselected album is shown in FIG. 2.

FIG. 2 shows a photo album 200 as displayed on a display device 202(e.g., a display integrated with and/or coupled to a computing device).The photo album 200 may be an example of a selected one of photo albums104 of FIG. 1. For example, the view of photo album 200 may be presentedresponsive to user selection of a cover photo for a corresponding photoalbum in user interface 100 of FIG. 1. A cover photo 204 may bepresented over the top of the photo album (e.g., along a header region206 of a displayed page)—or otherwise placed to indicate its associationwith the album. The cover photo may be the same photo as selected from aprior user interface screen (e.g., the view of the user interface 100 ofFIG. 1), and may be altered in order to conform to the spacing presentin the photo album 200. For example, the cover photo for a displayedalbum may be cropped, resized, stretched, and/or otherwise altered froman original photo in order to form an altered (e.g., cropped/resized)version of the photo that fits in the header region 206 of the album200. In this way, the cover photo may be altered in order to beappropriately dimensioned given the amount of screen space available forthe cover photo.

While the cover photo 204 may provide a single-photo representation ofthe album as a whole, the highlight photoset 208 may provide a fullerand/or otherwise different representation of the contents of the albumby indicating individuals who appear in the album. For example, eachphoto (e.g., 208 a-c) of the photoset 208 may include a cropped versionof a photo of the album that includes a recognized face of anindividual. The individuals included in the highlight photoset mayinclude individuals who appear in the album and are determined to have aclose connection to the user (e.g., the owner of the album) and/or thecontext of the album. For example, a relationship of recognizedindividuals in the album to the user may be determined based oninformation gathered from various sources, such as a contact historywith the user, social networking profiles for the user/individual,user-entered relationship data, and/or other information. The connectionof the individual to the context of the album may be identified based oninformation gathered from the same and/or additional or alternativesources, including information regarding the frequency of appearance ofthe individual in the photo album, a link between the date/eventcaptured by the album and the individual, and/or other information. Theidentification of connections between the individuals and the user/albumare described in more detail below with respect to FIGS. 4 and 5.

As illustrated, the highlight photoset 208 may at least partiallyoverlay the cover photo and be cropped such that only a face of arecognized individual is present in the cropped version of the photo.For example, only a single face (e.g., that of the recognizedindividual) may be present in the cropped version of the photo and noother faces may be present. As another example, one or more recognizedfaces of known individuals may be present in the cropped version of thephoto, and other features from the photo (e.g., landmarks, backgroundfaces/objects, etc.) may not be present in the cropped version of thephoto. In still another example, the cropped version may be cropped to apredetermined size/shape (e.g., the illustrated circle) and therecognized face of a known individual in the photo may be centeredwithin the predetermined size/shape of the cropped photo. In otherexamples, the highlight photoset 208 may occupy other regions thatindicate association with the album.

As further illustrated, an additional user interface element 210 may bedisplayed alongside the highlight photoset 208 with an indication ofadditional known individuals that are recognized as being included inthe photos of the album (e.g., 3 additional individuals in theillustrated example). The additional known individuals may includeremaining individuals having an affinity score higher than a threshold(e.g., the affinity score being determined as described in more detailbelow with respect to FIG. 3), where the individuals shown in thehighlight photoset include those individuals having the top affinityscores (e.g., the top three in this example, though the album mayinclude any number of top affinity scores in other examples). The numberof photos in the highlight photoset and/or the number of additionalknown individuals may be determined based on user preference settings,based on the size of the album (e.g., the number of photos in thealbum), the number of different recognized faces in the album, theamount of screen real estate available for the highlight photoset,and/or any other suitable factor.

The user interface element 210 may be selectable in order to view theremaining individuals. The view of the remaining individuals may bepresented in any suitable manner, including a listing of names or otheridentifiers of the additional individuals, a display of photo(s)including the additional individuals, a display of cropped versions ofphoto(s) including the additional individuals (e.g., matching theappearance, size, and/or shape of the highlight photoset 208), and/orany other presentation of data indicating the additional individuals.

The photos 212 of the album 200 (e.g., all of the photos of the album)may be presented below the cover photo 204 and highlight photoset 208.For example, a subset of the photos may be presented on a firstdisplayable page of the album 200, and additional photos of the album(e.g., all of the remaining photos of the album) may be presentedresponsive to scrolling, swiping, and/or otherwise navigating to adifferent page or view of the album. The photos 212 may be organized inany suitable manner, including by date, filename, size, affinity score,and/or other metric. Other views of the photos 212 may be presented,such as a layout having a hierarchical structure, showing the coverphoto, the highlight photos, the additional known individuals,thumbnails of the photos of the album, and/or original/larger-sizedversions of the photos from the album (relative to the thumbnails) insome sequential order (e.g., responsive to selection, scrolling, etc.).Any of the aspects of the album (e.g., cover photo, highlight photoset,photos of the album, etc.) may be viewable in different layouts/viewsresponsive to user input. Transitions between views may be performedusing an animation that bridges one view to a next view.

The photos 212 of the album may include original versions of the photosselected for the cover photo 204 and highlight photoset 208, as well asany or all other photos of the album. Each of the photos 212 may bepresented as thumbnail images (e.g., smaller sized versions of thephotos) that are selectable to view the original versions, for example.In this way, the cover photo and highlight photoset may berepresentative of all of the photos that are in the album by providing aquick view of a subset of the photos (e.g., and of relevant regions ofthe photos in that subset) that are likely to produce the highestemotional connection between the user and the album.

FIG. 3 is a flow chart of a method 300 for selecting representativephotos of an album based on determination and application of an affinityscore for individuals imaged in the photos of the album. Method 300 maybe performed by a single device, such as an end-user device (e.g., adevice on which photos are captured and/or displayed as albums) or aserver computing device(s) (e.g., a device for processing photos of analbum and returning information to another device so that the otherdevice may display the photo album with representative photos as a coverphoto and a highlight photoset). The end-user device may include amobile device (e.g., a smartphone, tablet, laptop, digital camera,and/or other mobile computing device), a personal computing device(e.g., a desktop computer), a home entertainment device (e.g., a videogame console, a video streaming device, and/or other consumer computingdevice). The server computing device may include a remote device from anend-user device, which may be connectable via a network (e.g., a cloudcomputing device). In some examples, the method 300 may be performed bya combination of two or more devices (e.g., where one or more processesof method 300 are performed by an end-user device that displays thephoto album and one or more other processes of method 300 are off-loadedto a server computing device, or server computing system with multipledevices, and a result returned to the end-user device).

At 302, the method includes receiving and/or capturing photos of analbum. For example, a user (e.g., an owner of the album and/or user ofthe device performing method 300) may select already captured photosand/or a category of already captured photos from a local or remotestorage device for inclusion in the album. The user may additionally oralternatively indicate that one or more photos that will be captured(e.g., the next [x] number of photos to be captured, or the next [x]number of photos that are captured that meet criteria such as date,time, location, and/or subject criteria) by an end-user device includinga camera are to be included in an album. In still other examples, one ormore photos may be automatically added to an album and/or an albumincluding one or more photos may be automatically created at or prior to302 based on features of the photos such as date, time, location, and/orsubject of the photos.

At 304, the method includes recognizing one or more faces in the photosof the album. A single device (e.g., an end-user device and/or a servercomputing device) may perform the facial recognition in some examples.In other examples, an end-user device may send one or more photos to aserver computing device for facial recognition. In such an example, theend-user device may identify the recognized faces based on data receivedfrom the server computing device. Facial recognition may be performed inany suitable manner, including but not limited to performing edgeanalysis or other object recognition techniques to locate faces and/orfacial features in the photos, and comparing the located faces/facialfeatures to those of known individuals. The known individuals mayinclude individuals whose faces/facials features are known to thesystem(s) performing the facial recognition (e.g., individuals whosefaces/facial features are stored in a database accessible by the facialrecognition system). In this way, the known individuals may be known tothe system, even if the known individuals are not known or well-known bythe user. In some examples, the recognition of faces may correspond tothe recognition that a feature in a photo is a face. In other words, aface may be recognized as a face and assigned an identifier (e.g.,unknown face x), even if the system has no other identifier for theface. Such an unknown face may also be compared to other faces in thealbum in order to determine if the unknown face appears multiple timeswithin the album and/or appears with the user and/or other knownindividuals (e.g., indicating a significance of that face relative toother faces that do not frequently appear in the album or with theuser/known individuals).

The method includes, for each known individual corresponding to arecognized face, determining an affinity score for that individual, asindicated at 306. As used herein, references to calculating, assigning,or otherwise generating a score, weight, or ranking may correspond toany suitable assessment of relative value. For example, the affinityscore may be based on a connection of the known individual to the user,as indicated at 308, and/or a connection of the known individual to thecontext of the album, as indicated at 310. Accordingly, scores, weights,and/or rankings applied to that individual may be an assessment of thatindividual's connection to the user and/or to the context of the albumrelative to other individuals. The connection of the individual to theuser may include a relationship of the individual to the user (e.g.,familial relationship, friendship, working relationship, mutualacquaintance, and/or other relationship) and may indicate a level ofemotional connection that the user may have toward that individual. Forexample, a best friend of a user may be given a higher value in auser-connection portion of an affinity score calculation than a distantcousin of the user.

The connection of the individual to the album context may include aconnection of the individual to an event being captured in the album, aconnection of the individual to the user that created/owns the album,and/or a connection of the individual to a date/location of the album.For example, an individual whose birthday occurs on the date of thealbum and/or who appears frequently in the album may be given a highervalue in an album context connection portion of an affinity scorecalculation than an individual who rarely appears in the album and hasno known connection to the date of the album. In some examples, anindividual may be given points for each connection to the user/albumcontext (e.g., with higher points given for stronger connections—such asthe best friend vs. distant cousin connection described above, where thebest friend is a stronger connection to the user than the distantcousin).

The points may be weighted based on an importance of the associatedconnection to the user (e.g., a strong relationship connection may beweighted higher than a strong album context connection, or vice versa;or a strong connection to a date of the album may be weighted higherthan a strong [e.g., frequent] presence of the individual in the album).The weighting and/or other aspects of affinity scoring may bepredetermined and/or set/modified by the user. Each score/value assessedmay be a contributing factor to the overall affinity score. By weightingdifferent contributing factors, known individuals that may otherwise beviewed as having a weak connection to the user, may be given an overallaffinity score that is comparable to an individual that has a strongconnection to the user in light of additional contributing connections.For example, although a coworker may initially be viewed as having aweaker connection to the user than a best friend, the coworker mayappear more frequently in the album than the best friend and/or in morephotos alongside the user than the best friend. The coworker may alsohave a connection to a date of the album (e.g., the album includesphotos captured on the coworker's birthday). These connections of thecoworker to the user and the album context may raise the coworker'soverall affinity score for that album to a value that is similar orhigher than the best friend. Additional features of the determination ofthe affinity score (e.g., sources of data used for determining theaffinity score) are described in more detail in FIGS. 4 and 5.

At 312, the method includes selecting known individuals based on theaffinity scores of those individuals. For example, the determination ofknown individuals (e.g., individuals whose faces are recognized in thephotos as corresponding to a name or other identifier of the individualin a database) may be used to form a set of known individuals that arepresent (e.g., captured) in the photo album. As each (or at least aportion) of the known individuals may be assigned an affinity score(e.g., calculated as described above and with respect to FIGS. 4 and 5),the affinity score may be used to determine how relevant each individualis to the user/album. Accordingly, those individuals with an affinityscore that meets certain criteria may be selected to be included in asubset of known individuals. For example, the criteria may include apredetermined number of known individuals having the highest affinityscores of the set of known individuals, and/or all or a predeterminednumber of known individuals having an affinity score that exceeds athreshold. Other criteria may be used to select the known individuals tobe included in the subset (e.g., known individuals having particularconnections to the user/album, etc.).

In some examples, rules associated with the assignment and/orapplication of affinity scores may include one or more machine learningmodels that a formed or evolve over time based on observed data. Forexample, the machine learning models may receive hand-tagged data and/orobserve end-users as the end-users make manual-override changes to facesthat are shown in album covers or highlight photosets. Based on suchdata, the machine learning models may adjust weighting, scores, and/orthe application thereof in order to generate affinity scores.

At 314, the method includes selecting a representative photo from thealbum for each selected individual of the subset. The representativephotos may be selected as the highest quality photos of the selectedindividuals, the photos having the most unobstructed and/or clearestview of the individual's face, and/or based on any other suitableparameter. For example, the quality of a given photo may be measuredusing one or more aspects such as sharpness (e.g., in or out of focus,with or without motion blur, etc.), exposure (e.g., brightness level,white balance, etc.), resolution, distortion, and/or other indicators ofquality. Such aspects may be compared with comparable aspects of otherphotos of a given selected individual and/or to a threshold in order toprovide an overall indication of photo quality.

As indicated at 316, at least one of the representative photos may beincluded in the album as the cover photo (e.g., selected to be used asthe cover photo). At least one of the representative photos may beincluded in the highlight photoset, as indicated at 318. In someexamples, the photo selected for the cover photo may also be included inthe highlight photoset. The cover photo may be selected from therepresentative photos based on the affinity score of the individual(s)included in that photo, the number of individuals recognized in thatphoto, and/or the quality, size, content, date, filename, or otherfeature of that photo relative to other photos in the selectedrepresentative photos. As described above, the representative photosused in the highlight photoset may be selected based on the associatedaffinity scores of those photos.

Method 300 optionally includes presenting the cover photo and highlightphotoset, as indicated at 320. For example, if method 300 is performedby an end-user device, the end-user device may present the cover photoand highlight photoset (e.g., where the highlight photoset includescropped representative photos, as indicated at 322 and illustrated inFIG. 2). If method 300 is performed by a server computing device (e.g.,a device remote from the end-user device), the method may includesending the cover photo and highlight photoset (or identifiers thereof)to a remote device (e.g., the end-user device) for presentation, asindicated at 324. In some examples, method 300 may be initializedautomatically upon creation of an album and/or responsive to user inputrequesting album creation and/or selection of representative photos fora created album.

FIGS. 4 and 5 illustrate information sources for retrieving data used indetermining an affinity score (e.g., the affinity score discussed withrespect to method 300 of FIG. 3). The information sources and associatedretrieved data described with respect to FIGS. 4 and 5 are exemplary,and additional or alternative data may be retrieved and evaluated inorder to determine the above-described affinity score. Furthermore, anycombination of the described data may be used in calculation of anaffinity score (e.g., based on the information that is available for agiven user/known individual). FIG. 4 is a flow chart of a method 400 forretrieving and evaluating data relating to a user, a known individual(e.g., a recognized individual in a photo album of the user), and/or aphoto album of the user. For example, method 400 may be performed atand/or as part of process 306 of method 300 illustrated in FIG. 3.

At 402, method 400 includes querying a social networking service. Forexample, the user, the known individual, and/or individuals relating tothe known individual may be users of one or more social networkingservices and/or maintain a profile or account at a social networkingservice. The social networking activity/profile/account may indicatefriends/relationships between the user and other individuals, groups,likes, follows, posts, and/or other information indicating useraffinities. The information may be compared to that of known individualsin order to determine overlaps indicating similar interests/connections.The social networking query may be used to determine messaging historybetween a known/recognized individual and the user (e.g., messaging viathe social networking service and/or another service), as indicated at404. For example, a frequency and/or number of messaging between theuser and the individual may indicate an associated relationship (e.g.,where a higher frequency/greater amount of messaging indicates a closerrelationship than a lower frequency/amount of messaging). As will becomeapparent in the following description, such data may only serve as oneindicator of relationship, and other data (e.g., the other datadescribed in method 400) may be combined with these indicators to forman overall relationship score (e.g., a score utilized in determining theaffinity score, which may be based on a relationship score and an albumcontext score).

As indicated at 406, the social networking service may be queried todetermine mentions of the individual and/or user in one another's posts,profiles, and/or activity feeds for the social networking service. Forexample, a user may often tag a known individual in status updates orlocation check-ins, indicating that the user is often socializing withthe known individual. Each mention may be tallied in order to indicate alikely affinity of the user to that known individual (e.g., where agreater number of mentions indicates a likely stronger connectionbetween the user and the known individual). Mentions may also beevaluated from other social networking profiles/accounts (e.g., ofmutual friends between the user and the known individual), as a thirdparty may post and/or check-in with both the user and the knownindividual, even when the user/individual do not include suchposts/check-ins on their own accounts/profiles.

As indicated at 408, the social networking service may be queried toidentify overlapping events and/or interests between the user and theknown individual. For example, a user and a known individual with manycommon interests and/or who attend the same events may be closer thanusers and individuals with fewer common interests and/or who attendfewer and/or no events together. The overlapping events may alsoindicate a context of the album and a connection of the individualthereto. For example, an album including photos that were captured onthe same day as an event that both the user and the individual attendedmay be indicative of a connection between the individual and the contextof the album.

As indicated at 410, the social networking service may be queried toidentify the presence and/or frequency of appearance of the individualand/or user in one another's photo albums (e.g., the photo albums hostedby and/or linked to via the social networking service). For example, auser and a known individual who appear in one another's photosfrequently and/or appear in a large number of one another's photos mayhave a closer relationship than a user/individual who appear in oneanother's photos less frequently and/or appear in a fewer number of oneanother's photos.

The frequency/number of appearances in photo albums may also indicate aconnection between the individual and the context of the photo albumbeing represented (e.g., by performing method 300 of FIG. 3). Forexample, a high frequency of appearances of the individual in the photoalbum being represented may be one indicator of a higher connection ofthe individual to the photo album (relative to individuals who appearless frequently in the album). The frequency of appearance of theindividual with the user and/or other known individuals in the album mayindicate a higher connection of the individual to the photo album and/orthe user (relative to individuals who appear with the user lessfrequently in the album). As discussed above, the frequency ofappearance may not be the only metric evaluated when determiningrelevance of an individual to a context of the album and/or whendetermining an overall affinity score for the individual. For example, aclose friend may be capturing a majority of the photos of the album, andthus appear less frequently in the album, but may still be important tothe context of the album (e.g., if the album is associated with thatindividual's birthday or a trip taken by that individual and the user)and/or the user (e.g., if that individual is the user's best friend).Thus, while the frequency of appearance in the album may be low for agiven individual, the individual may still have an overall high affinityscore due to the contribution of connections identified by other data.

At 412, the method includes querying a mobile device and/or personalcomputing device of the user. For example, this querying may be used todetermine call and/or messaging (e.g., SMS/MMS, email, etc.) historybetween the known/recognized individual and the user, as indicated at414. As described above with respect to the social networking servicemessaging, frequent and/or large amounts of messaging between the userand the individual may indicate a closer relationship than less frequentand/or smaller amounts of messaging.

As indicated at 416, the mobile device and/or personal computing devicemay be queried to identify events involving the individual (e.g., basedon a calendar of the user and/or another event trackingservice/application running on the user's device). For example, anindividual that is included in a large number of events in the user'scalendar and/or an individual who has overlapping events with the user(e.g., as determined by querying the social networking profile of theindividual) may be closer to the user than other individuals who do nothave as many overlapping events. The events of the user may also beevaluated to determine a context of an album (e.g., if thedate/time/location of the events overlap with the date/time/location ofthe album and/or if descriptions of the album and events overlap).

As indicated at 418, the mobile device and/or personal computing devicemay be queried in order to identify the presence of the individualand/or the individual with the user in the user's albums (e.g., otheralbums than the album for which representative photos are being selectedaccording to method 300 of FIG. 3, or the album being represented). Forexample, a high frequency of appearance of the individual in the user'salbums and/or a high frequency of appearance of the individual with theuser in the user's albums may indicate a close relationship between theuser and the individual. A high frequency of appearance of theindividual in the album being represented may indicate a connection ofthe individual to the context of the album. In determining frequency ofappearance, appearances with the user may be ranked/weighted higher thanappearances without the user in some examples. In other examples, allappearances may be weighted/ranked the same as one another.

As indicated at 420, the mobile device and/or personal computing devicemay be queried in order to identify web browsing history of the userthat relates to the individual. For example, web browsing history may beused to indicate interests of the user (e.g., favorite movies,television shows, music, etc.) and individuals associated with thoseinterests (e.g., favorite actors/actresses, singers, chefs, etc.). Suchinterests may also be determined by examining social networking activityof the user (e.g., following or liking a celebrity on a socialnetworking service). Photos with individuals associated with theinterests (e.g., celebrities) may be weighted/ranked heavily whendetermining an affinity score for those individuals, as such individualsmay not appear in the user's other photos/everyday life, but may holdhigh significance to the user, indicating a memorable occurrence withinthe context of the photo album (e.g., during a vacation). The webbrowsing history may also be used to determine a context of an album.For example, a user may perform a large number of web searches relatingto planning a trip prior to creating the album. An overlap between theweb searches and the location of the photos of the album may indicatethat the album relates to the planned trip.

As indicated at 422, the mobile device and/or personal computing devicemay be queried in order to identify a user location (e.g., a currentlocation and/or locations observed over time/frequented by the user toform a travel history for that user). For example, locations that arefrequented by the user may be compared to locations frequented by theindividual and/or locations at which photos of the album were taken inorder to determine a relationship to the individual and/or a context ofthe album.

At 424, the method includes identifying a relationship of the individualto the user based on data received from the queries. Any suitablecomputation of the queried data may be used to determine therelationship. For example, each connection between the user and theindividual may be assigned a value (e.g., indicating a strength of thatconnection—for example, the presence of an individual in any photo ofthe user may be a connection between the user and the individual, but afirst individual that is in a first, higher number of the user's photosmay be assigned a higher value for that connection than a secondindividual that is in a second, lower number of the user's photos).Furthermore, each connection may be assigned a weight indicating theimportance of that connection to the overall relationship score and/orto the overall affinity score. For example, a connection between anindividual and a search history of the user (e.g., indicating that theindividual is a favorite actor/singer/etc. of the user) may be weightedhigher than a connection between an individual and user's location.

At 426, the method includes identifying a connection between theindividual and an album context based on the data received from thequeries. As described above with respect to the relationshipdetermination, each connection of the individual to the album contextmay be assigned a value and weighted. The values and weights applied tothese connections (e.g., for the relationship determination and thealbum context determination) may be predetermined and/orset/configurable by user input. In some examples, the user may identifya weighting to be used on a per-album basis. For example, a user mayconfigured an album relating to a birthday such that an individual'sconnection to the album context is weighted more heavily than for thatalbum than for an album relating to a vacation. The connections to thealbum context and to the user may be combined and used to determine anaffinity score as described above with respect to method 300 of FIG. 3.

FIG. 5 shows example block diagrams of devices that may be queried todetermine connections for an individual. Diagram 500 a shows devicesused for determining connections of an individual to a user (e.g., arelationship between the user and the individual), and diagram 500 bshows devices used for determining connections of an individual to auser album (e.g., a connection between the individual and the albumcontext). Block diagrams 500 a and 500 b are exemplary, and any suitabledevice may be queried to gather any suitable data for determiningconnections for an individual. For example, devices/data described indiagram 500 a for relationship determination may additionally oralternatively be used for album context connection determination, andvice versa.

For determining a relationship between the user and the individual, amobile device 502 of the user may be queried to identify messaging(e.g., SMS messaging) and/or phone conversations between the user andthe individuals. The mobile device 502 may also be queried for webbrowsing information relating to the individual. A personal computingdevice 504 of the user may be queried to identify further web browsinginformation relating to the individual, as well as photo albums local tothe machine that may include photos of the individual. For example, auser may backup his/her photos to a personal computing device, so thepersonal computing device may serve as an archive for the user's photos.The web browsing information may also include web-based messaging (e.g.,video conferencing, instant messaging, emailing, etc.) that may occurbetween the user and the individual.

A server computing device 506 may be queried to identify socialnetworking contacts, third-party-hosted web browsing information,cloud/network-stored photo albums, and/or social networking messages.Server computing device 506 may be representative of any number ofserver computing devices, such as the servers for each social networkingservice to which the user belongs.

For determining an album context for a user's album, the user's mobiledevice 502 may be queried to identify calendar data (e.g.,date/time/location of significant events for the user), locationinformation, and/or web browsing data. The user's personal computingdevice 504 may be queried to identify further web browsing data,calendar information (e.g., as discussed above with respect to the photoalbums, the personal computing device 504 may archive calendarinformation for evaluation), and/or user entry (e.g., a user may specifya context of the album when creating the album). The server computingdevice(s) 506 may be used to perform object recognition in the photos(e.g., photos that include a birthday cake may indicate that the albumis related to a birthday party; identified landmarks may indicate thatthe album is related to a vacation and/or may indicate a location atwhich the photos were taken). The server computing device(s) may also bequeried for web browsing information, calendar data (e.g., a cloud-basedcalendar for the user), and social networking profiles of otherindividuals (e.g., to determine overlapping events for individuals andthe album). For example, the album may take place on an individual'sbirthday, but only the individual may have the birthday identified onhis/her social networking profile. This information may help to identifythe context of the album as relating to the individual's birthday (andidentify the individual's connection to the album), even if thatbirthday is not an event on the user's calendar.

In the above examples, various data may be used to calculate an affinityof a recognized individual to a user. The affinity may be indicative ofthe individual's relationship to both the user and the album context. Byusing such varied information sources, the system may make a moreinformed determination of relationships between the individual and theuser/album context. Furthermore, an identified album context mayindicate connections other than the relationship between the user andthe individual. A connection between an individual and such an albumcontext may provide more insight into the overall importance of theindividual to that album, and result in representative photo selectionsthat have more significance to the user and the album than photoselections that only factor user/individual relationships based on a fewmetrics. Representative photos that are significant to both the user andthe album context may assist the user in remembering the context of thealbum and emotionally connecting to the album, even before viewing allof the photos in the album.

In some embodiments, the methods and processes described herein may betied to a computing system of one or more computing devices. Inparticular, such methods and processes may be implemented as acomputer-application program or service, an application-programminginterface (API), a library, and/or other computer-program product.

FIG. 6 schematically shows a non-limiting embodiment of a computingsystem 600 that can enact one or more of the methods and processesdescribed above. Computing system 600 is shown in simplified form.Computing system 600 may take the form of one or more personalcomputers, server computers, tablet computers, home-entertainmentcomputers, network computing devices, gaming devices, mobile computingdevices, mobile communication devices (e.g., smart phone), and/or othercomputing devices.

Computing system 600 includes a logic machine 602 and a storage machine604. Computing system 600 may optionally include a display subsystem606, input subsystem 608, communication subsystem 610, and/or othercomponents not shown in FIG.6.

Logic machine 602 includes one or more physical devices configured toexecute instructions. For example, the logic machine may be configuredto execute instructions that are part of one or more applications,services, programs, routines, libraries, objects, components, datastructures, or other logical constructs. Such instructions may beimplemented to perform a task, implement a data type, transform thestate of one or more components, achieve a technical effect, orotherwise arrive at a desired result.

The logic machine may include one or more processors configured toexecute software instructions. Additionally or alternatively, the logicmachine may include one or more hardware or firmware logic machinesconfigured to execute hardware or firmware instructions. Processors ofthe logic machine may be single-core or multi-core, and the instructionsexecuted thereon may be configured for sequential, parallel, and/ordistributed processing. Individual components of the logic machineoptionally may be distributed among two or more separate devices, whichmay be remotely located and/or configured for coordinated processing.Aspects of the logic machine may be virtualized and executed by remotelyaccessible, networked computing devices configured in a cloud-computingconfiguration.

Storage machine 604 includes one or more physical devices configured tohold instructions executable by the logic machine to implement themethods and processes described herein. When such methods and processesare implemented, the state of storage machine 604 may betransformed—e.g., to hold different data.

Storage machine 604 may include removable and/or built-in devices.Storage machine 604 may include optical memory (e.g., CD, DVD, HD-DVD,Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM,etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive,tape drive, MRAM, etc.), among others. Storage machine 604 may includevolatile, nonvolatile, dynamic, static, read/write, read-only,random-access, sequential-access, location-addressable,file-addressable, and/or content-addressable devices.

It will be appreciated that storage machine 604 includes one or morephysical devices. However, aspects of the instructions described hereinalternatively may be propagated by a communication medium (e.g., anelectromagnetic signal, an optical signal, etc.) that is not held by aphysical device for a finite duration.

Aspects of logic machine 602 and storage machine 604 may be integratedtogether into one or more hardware-logic components. Such hardware-logiccomponents may include field-programmable gate arrays (FPGAs), program-and application-specific integrated circuits (PASIC/ASICs), program- andapplication-specific standard products (PSSP/ASSPs), system-on-a-chip(SOC), and complex programmable logic devices (CPLDs), for example.

The terms “module,” “program,” and “engine” may be used to describe anaspect of computing system 600 implemented to perform a particularfunction. In some cases, a module, program, or engine may beinstantiated via logic machine 602 executing instructions held bystorage machine 604. It will be understood that different modules,programs, and/or engines may be instantiated from the same application,service, code block, object, library, routine, API, function, etc.Likewise, the same module, program, and/or engine may be instantiated bydifferent applications, services, code blocks, objects, routines, APIs,functions, etc. The terms “module,” “program,” and “engine” mayencompass individual or groups of executable files, data files,libraries, drivers, scripts, database records, etc.

It will be appreciated that a “service”, as used herein, is anapplication program executable across multiple user sessions. A servicemay be available to one or more system components, programs, and/orother services. In some implementations, a service may run on one ormore server-computing devices.

When included, display subsystem 606 may be used to present a visualrepresentation of data held by storage machine 604. This visualrepresentation may take the form of a graphical user interface (GUI). Asthe herein described methods and processes change the data held by thestorage machine, and thus transform the state of the storage machine,the state of display subsystem 606 may likewise be transformed tovisually represent changes in the underlying data. Display subsystem 606may include one or more display devices utilizing virtually any type oftechnology. Such display devices may be combined with logic machine 602and/or storage machine 604 in a shared enclosure, or such displaydevices may be peripheral display devices.

When included, input subsystem 608 may comprise or interface with one ormore user-input devices such as a keyboard, mouse, touch screen, or gamecontroller. In some embodiments, the input subsystem may comprise orinterface with selected natural user input (NUI) componentry. Suchcomponentry may be integrated or peripheral, and the transduction and/orprocessing of input actions may be handled on- or off-board. Example NUIcomponentry may include a microphone for speech and/or voicerecognition; an infrared, color, stereoscopic, and/or depth camera formachine vision and/or gesture recognition; a head tracker, eye tracker,accelerometer, and/or gyroscope for motion detection and/or intentrecognition; as well as electric-field sensing componentry for assessingbrain activity.

When included, communication subsystem 610 may be configured tocommunicatively couple computing system 600 with one or more othercomputing devices. Communication subsystem 610 may include wired and/orwireless communication devices compatible with one or more differentcommunication protocols. As non-limiting examples, the communicationsubsystem may be configured for communication via a wireless telephonenetwork, or a wired or wireless local- or wide-area network. In someembodiments, the communication subsystem may allow computing system 600to send and/or receive messages to and/or from other devices via anetwork such as the Internet.

Another example provides computing device for selecting one or morerepresentative photos of an album comprising one or more photos, thecomputing device including a display, a logic device, and a storagedevice storing instructions executable by the logic device to recognizefaces in each photo of the album, each recognized face corresponding toa known individual, for each known individual recognized in one or morephotos of the album, determine an affinity score for that knownindividual, the affinity score indicating a connection of that knownindividual to a) a user of the computing device and b) a contextcorresponding to the album, select one or more known individuals basedon the affinity score to be included in a subset of the knownindividuals recognized in one or more photos of the album, for each ofthe one or more known individuals in the subset, select a representativephoto from the album including a recognized face of that knownindividual to be included in one or more of a highlight photoset and acover photo for the album, and present, on the display, the cover photoincluding a first one of the representative photos, and the highlightphotoset including at least a second one of the representative photos.In such an example, the highlight photoset may additionally oralternatively include a cropped version of at least the second one ofthe representative photos. In such an example, the cropped version ofthe second one of the representative photos may additionally oralternatively include a single face, the single face being therecognized face corresponding to a known individual in the subset ofknown individuals. In such an example, the cropped version of at leastthe second one of the representative photos may additionally oralternatively be presented as an overlay at least partially overlayingthe cover photo. In such an example, the affinity score may additionallyor alternatively be determined based on a relationship of the knownindividual with the user, the relationship being determined based on oneor more of user selection and examination of one or more of a) socialnetworking activity, b) a social network profile, c) messaging historyfor the user, d) travel history of the user, and e) a frequency ofpresence of the known individual in other photos associated with theuser. In such an example, the context of the album may additionally oralternatively be determined based on one or more of a) objectrecognition of objects in the one or more photos, b) identification ofdate, time, or location of the one or more photos, and c) a profile ofone or more of the user and the known individuals. In such an example,for each known individual, selecting the representative photo from thealbum including a recognized face of that known individual mayadditionally or alternatively include selecting the representative photobased on a quality of the recognized face of that known individual ineach photo of the album that includes the known individual. In such anexample, the cover photo may additionally or alternatively include oneor more of a cropped version and a resized version of the first one ofthe representative photos. In such an example, the resized version ofthe first one of the representative photos may additionally oralternatively be presented as the cover photo in a first user interfaceview presented via the display and wherein the cropped version of thefirst one of the representative photos is presented as the cover photoin a second user interface view presented via the display. In such anexample, the highlight photoset may additionally or alternatively bepresented as an overlay over at least part of the cropped version of thefirst one of the representative photos presented in the second userinterface view. Any or all of the above-described examples may becombined in any suitable manner in various implementations.

Another example provides, on a computing device, a method for selectingrepresentative photos of a photo album, the method including identifyingfaces in each photo of the photo album, each identified facecorresponding to a known individual, for each known individualrecognized in one or more photos of the photo album, identifying anaffinity score for that known individual, the affinity score indicatinga connection of that known individual to a) a user of the computingdevice and b) a context corresponding to the photo album, selecting oneor more known individuals based on the affinity score to be included ina subset of the known individuals recognized in one or more photos ofthe photo album, for each of the one or more known individuals in thesubset, selecting a representative photo from the photo album includingan identified face of that known individual to be included in one ormore of a highlight photoset and a cover photo for the photo album, andpresenting, on the display the cover photo including a first one of therepresentative photos, and the highlight photoset including at least asecond one of the representative photos. In such an example, identifyingfaces in each photo of the photo album may additionally or alternativelyinclude sending one or more photos from the photo album to a remotecomputing device and receiving an indication of recognized faces in theone or more photos from the remote computing device. In such an example,identifying the affinity score for the known individual may additionallyor alternatively include receiving the affinity score from a remotecomputing device. In such an example identifying the affinity score forthe known individual includes receiving information from a remotecomputing device indicating one or more of the connection of the knownindividual to the user of the computing device and the connection of theknown individual to the context corresponding to the photo album. Insuch an example, the method may additionally or alternatively furtherinclude cropping one or more of the representative photos selected forthe one or more known individuals in the subset, and wherein thehighlight photoset includes a cropped version of at least the second oneof the representative photos. Any or all of the above-described examplesmay be combined in any suitable manner in various implementations.

Another example provides a first computing device for selecting one ormore representative photos of an album comprising one or more photos,the computing device including a network interface, a logic device, anda storage device storing instructions executable by the logic device toreceive, from a second computing device via the network interface, oneor more photos of the album, recognize faces in each received photo ofthe album, each recognized face corresponding to a known individual, foreach known individual recognized in one or more of the received photosof the album, determine an affinity score for that known individual, theaffinity score indicating a connection of that known individual to a) auser of the second computing device and b) a context corresponding tothe album, select one or more known individuals based on the affinityscore to be included in a subset of the known individuals recognized inone or more photos of the album, for each of the one or more knownindividuals in the subset, select a representative photo from the albumincluding a recognized face of that known individual to be included inone or more of a highlight photoset and a cover photo for the album, andtransmit, to the second computing device via the network interface fordisplay at the second computing device the cover photo including a firstone of the representative photos, and the highlight photoset includingat least a second one of the representative photos. In such an example,the network interface may additionally or alternatively include awireless network interface. In such an example, determining the affinityscore for the known individual may additionally or alternatively includequerying a remote computing device for user information relating to theknown individual. In such an example, the affinity score mayadditionally or alternatively be determined based on a relationship ofthe known individual with the user, the relationship being determinedbased on one or more of user selection and examination of one or more ofa) social networking activity, b) a social network profile, c) messaginghistory for the user, d) travel history of the user, and e) a frequencyof presence of the known individual in other photos associated with theuser. In such an example, the context of the album may additionally oralternatively be determined based on one or more of a) objectrecognition of objects in the one or more photos, b) identification ofdate, time, or location of the one or more photos, and c) a profile ofone or more of the user and the known individuals. Any or all of theabove-described examples may be combined in any suitable manner invarious implementations.

It will be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated and/ordescribed may be performed in the sequence illustrated and/or described,in other sequences, in parallel, or omitted. Likewise, the order of theabove-described processes may be changed.

The subject matter of the present disclosure includes all novel andnon-obvious combinations and sub-combinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

1. A computing device for selecting one or more representative photos ofan album comprising one or more photos, the computing device comprising:a display; a logic device; and a storage device storing instructionsexecutable by the logic device to: recognize faces in each photo of thealbum, each recognized face corresponding to a known individual; foreach known individual recognized in one or more photos of the album,determine an affinity score for that known individual, the affinityscore indicating a connection of that known individual to a) a user ofthe computing device and b) a context corresponding to the album; selectone or more known individuals based on the affinity score to be includedin a subset of the known individuals recognized in one or more photos ofthe album; for each of the one or more known individuals in the subset,select a representative photo from the album including a recognized faceof that known individual to be included in one or more of a highlightphotoset and a cover photo for the album; and present, on the display:the cover photo including a first one of the representative photos, andthe highlight photoset including at least a second one of therepresentative photos.
 2. The computing device of claim 1, wherein thehighlight photoset includes a cropped version of at least the second oneof the representative photos.
 3. The computing device of claim 2,wherein the cropped version of the second one of the representativephotos includes a single face, the single face being the recognized facecorresponding to a known individual in the subset of known individuals.4. The computing device of claim 2, wherein the cropped version of atleast the second one of the representative photos is presented as anoverlay at least partially overlaying the cover photo.
 5. The computingdevice of claim 1, wherein the affinity score is determined based on arelationship of the known individual with the user, the relationshipbeing determined based on one or more of user selection and examinationof one or more of a) social networking activity, b) a social networkprofile, c) messaging history for the user, d) travel history of theuser, and e) a frequency of presence of the known individual in otherphotos associated with the user.
 6. The computing device of claim 1,wherein the context of the album is determined based on one or more ofa) object recognition of objects in the one or more photos, b)identification of date, time, or location of the one or more photos, andc) a profile of one or more of the user and the known individuals. 7.The computing device of claim 1, wherein for each known individual,selecting the representative photo from the album including a recognizedface of that known individual comprises selecting the representativephoto based on a quality of the recognized face of that known individualin each photo of the album that includes the known individual.
 8. Thecomputing device of claim 1, wherein the cover photo includes one ormore of a cropped version and a resized version of the first one of therepresentative photos.
 9. The computing device of claim 8, wherein theresized version of the first one of the representative photos ispresented as the cover photo in a first user interface view presentedvia the display and wherein the cropped version of the first one of therepresentative photos is presented as the cover photo in a second userinterface view presented via the display.
 10. The computing device ofclaim 9, wherein the highlight photoset is presented as an overlay overat least part of the cropped version of the first one of therepresentative photos presented in the second user interface view. 11.On a computing device, a method for selecting representative photos of aphoto album, the method comprising: identifying faces in each photo ofthe photo album, each identified face corresponding to a knownindividual; for each known individual recognized in one or more photosof the photo album, identifying an affinity score for that knownindividual, the affinity score indicating a connection of that knownindividual to a) a user of the computing device and b) a contextcorresponding to the photo album; selecting one or more knownindividuals based on the affinity score to be included in a subset ofthe known individuals recognized in one or more photos of the photoalbum; for each of the one or more known individuals in the subset,selecting a representative photo from the photo album including anidentified face of that known individual to be included in one or moreof a highlight photoset and a cover photo for the photo album; andpresenting, on the display: the cover photo including a first one of therepresentative photos, and the highlight photoset including at least asecond one of the representative photos.
 12. The method of claim 11,wherein identifying faces in each photo of the photo album includessending one or more photos from the photo album to a remote computingdevice and receiving an indication of recognized faces in the one ormore photos from the remote computing device.
 13. The method of claim11, wherein identifying the affinity score for the known individualincludes receiving the affinity score from a remote computing device.14. The method of claim 11, wherein identifying the affinity score forthe known individual includes receiving information from a remotecomputing device indicating one or more of the connection of the knownindividual to the user of the computing device and the connection of theknown individual to the context corresponding to the photo album. 15.The method of claim 11, further comprising cropping one or more of therepresentative photos selected for the one or more known individuals inthe subset, and wherein the highlight photoset includes a croppedversion of at least the second one of the representative photos.
 16. Afirst computing device for selecting one or more representative photosof an album comprising one or more photos, the computing devicecomprising: a network interface; a logic device; and a storage devicestoring instructions executable by the logic device to: receive, from asecond computing device via the network interface, one or more photos ofthe album; recognize faces in each received photo of the album, eachrecognized face corresponding to a known individual; for each knownindividual recognized in one or more of the received photos of thealbum, determine an affinity score for that known individual, theaffinity score indicating a connection of that known individual to a) auser of the second computing device and b) a context corresponding tothe album; select one or more known individuals based on the affinityscore to be included in a subset of the known individuals recognized inone or more photos of the album; for each of the one or more knownindividuals in the subset, select a representative photo from the albumincluding a recognized face of that known individual to be included inone or more of a highlight photoset and a cover photo for the album; andtransmit, to the second computing device via the network interface fordisplay at the second computing device: the cover photo including afirst one of the representative photos, and the highlight photosetincluding at least a second one of the representative photos.
 17. Thefirst computing device of claim 16, wherein the network interfaceincludes a wireless network interface.
 18. The first computing device ofclaim 16, wherein determining the affinity score for the knownindividual includes querying a remote computing device for userinformation relating to the known individual.
 19. The first computingdevice of claim 16, wherein the affinity score is determined based on arelationship of the known individual with the user, the relationshipbeing determined based on one or more of user selection and examinationof one or more of a) social networking activity, b) a social networkprofile, c) messaging history for the user, d) travel history of theuser, and e) a frequency of presence of the known individual in otherphotos associated with the user.
 20. The first computing device of claim16, wherein the context of the album is determined based on one or moreof a) object recognition of objects in the one or more photos, b)identification of date, time, or location of the one or more photos, andc) a profile of one or more of the user and the known individuals.